Skip to content

PERF: Series.to_numpy with float dtype and na_value=np.nan #52430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 6, 2023
Merged

PERF: Series.to_numpy with float dtype and na_value=np.nan #52430

merged 3 commits into from
Apr 6, 2023

Conversation

lukemanley
Copy link
Member

Avoids filling na values with np.nan if the array is already a numpy float dtype. (DataFrame has a similar optimization in BlockManager.as_array)

import pandas as pd
import numpy as np

ser = pd.Series(np.random.randn(10_000_000))

%timeit ser.to_numpy(dtype="float64", na_value=np.nan, copy=False)

# 22.5 ms ± 2.56 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)          -> main
# 1.97 µs ± 24.5 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)   -> PR

ASV added:

       before           after         ratio
     [3ce07cb4]       [020fb14a]
     <main>           <series-to-numpy-float-nan>
-     1.02±0.02ms      2.12±0.03μs     0.00  series_methods.ToNumpy.time_to_numpy_float_with_nan

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

@lukemanley lukemanley added the Performance Memory or execution speed performance label Apr 5, 2023
@topper-123 topper-123 added the Series Series data structure label Apr 5, 2023
@topper-123 topper-123 added this to the 2.1 milestone Apr 5, 2023
topper-123
topper-123 approved these changes Apr 5, 2023
View reviewed changes
Copy link
Contributor

@topper-123 topper-123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@topper-123
Copy link
Contributor

Actually, what about FloatDtype(64|32)?

@lukemanley
Copy link
Member Author

Actually, what about FloatDtype(64|32)?

EA dtypes dispatch to their own implementation of to_numpy. In the case of FloatDtype(64|32), I'm not sure there is anything to be done. BaseMaskedArray.to_numpy needs to combine the data array and the mask array when NAs exist.

topper-123
topper-123 approved these changes Apr 5, 2023
View reviewed changes
Copy link
Contributor

@topper-123 topper-123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@mroeschke mroeschke merged commit 0e857ff into pandas-dev:main Apr 6, 2023
@mroeschke
Copy link
Member

Thanks @lukemanley

topper-123 pushed a commit to topper-123/pandas that referenced this pull request Apr 6, 2023
@lukemanley @topper-123
PERF: Series.to_numpy with float dtype and na_value=np.nan (pandas-de…
3ff1303 
…v#52430)

* PERF: Series.to_numpy with dtype=float and na_value=np.nan

* whatsnew

* add space
@lukemanley lukemanley deleted the series-to-numpy-float-nan branch April 18, 2023 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mroeschke mroeschke mroeschke approved these changes

@topper-123 topper-123 topper-123 approved these changes

Assignees
No one assigned
Labels
Performance Memory or execution speed performance Series Series data structure
Projects
None yet
Milestone
2.1
Development

Successfully merging this pull request may close these issues.
3 participants
@lukemanley @topper-123 @mroeschke